Computer animation has been used to produce computer-generated pictures associated with various characteristics. Usually a computer animation is used to produce a moving animational system. As the users speak, their mouths move, but the movement of the mouths of the speakers and their speech has not been synchronized. This does not bother the viewer, however, since it appears to be a cartoon; and is not intended to be accurate.
The inventors of the present invention recognized that usual computer animation does not provide a sufficiently accurate picture of a user speaking to allow it to be used as a facsimile of that user speaking. That is, under the current state of the art, the inventors of the present invention recognized that a viewer of the computer animation would never be fooled into believing that the computer animation was real. They set about trying to find a way to solve this problem.
The inventors recognized, for the first time, that morphing technology could be used to simulate moving facial characteristics. Morphing technology is well known in the art: it is used to simulate a continuous change from a first image of a first object into an image of a second object. For example, it is easy to morph as apple into an orange. While one is looking at the apple, one sees its characteristics gradually change. It gradually assumes the shape of the orange, and also gradually assumes the color of the orange.
Morphing is well known, but a brief explanation of its operation will be given here anyway. Morphing involves transforming a first object, an "original object" into a second object, a "destination object". The computer takes the original object and the destination object, and maps various points thereof. These points define the shape and contour of both objects as well as the colors at the various points. Morphing can be carried out using a number of different techniques. For simplicity, we can assume that a small number of points, e.g., 16 points are used.
The morphing process is then calculated in advance: an interim point between the two objects is calculated, and then interim points between those objects are calculated. These interim points can be any points between the two objects. This provides a plurality of images, each image differing from the previous image by only a small amount, and each image incrementally closer to the destination image. By providing a number of images, over an amount of time, the difference between each two adjacent images is very small. The viewer sees the illusion of transformation from one image to another, and thus the user sees a continuously-varying image that changes gradually from the original image to the destination image. It appears as though the apple changes into the orange.
The inventors of the present invention were the first to realize that such morphing technology could be used to simulate an image of a human body part moving between a first position and a second position for computer animation purposes.
One specific aspect of the present invention is the use of morphing technology to change facial image characteristics in a way to simulate the characteristics of speech. To do this, the inventors of the present invention developed a plurality of tools which changed human facial expression in accordance with speech to be spoken. The detailed aspects will be described herein.
It is therefore an object of the present invention to provide a system and method which changes facial expressions of a user's body part, preferably a user's face, in a way that associates those facial expressions with speech.